Sp 2n (F q 2 )-IN VARIANTS IN IRREDUCIBLE UNIPOTENT 
REPRESENTATIONS OF Sp 4n (F g ) 



LEI ZHANG 



Abstract. We show that for any irreducible representation of 
Sp 4n (F g ), the subspace of all its Sp 2ll (F 9 2)-invariants is at most 
one-dimensional. In terms of Lusztig symbols, we give a com- 
plete list of irreducible unipotent representations of Sp 4 „(F 9 ) which 
have a nonzero Sp 2 „(F 9 2)-invariant and, in particular, we prove 
that every irreducible unipotent cuspidal representation has a one- 
dimensional subspace of Sp 2n (F„2 )-invariants. As an application, 
we give an elementary proof of the fact that the unipotent cuspidal 
representation is defined over Q, which was proved by Lusztig in 
[L02]. 
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1. Introduction 

Let G be a reductive group defined over a field k and 9 be an invo- 
lution of G defined over k. Denote by H the subgroup of all ^-fixed 
points in G, called a symmetric subgroup of G. One of the classical 
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problems in harmonic analysis on symmetric spaces G(k)/H(k) is to 
give a formula for the dimension of the space Hornet) (71", IndSnl) for 
each irreducible representation 7r of G(k) and the trivial representation 
1 of H(k), or equivalently, for the dimension of the space of all H(k)- 
invariant linear functionals on the space of 7r. By Frobenius reciprocity, 
this dimension is equal to the dimension dimn H( - k ^ of the subspace of 
H (A;)-invariants. 

For example, let A; be a non-archimedean field of characteristic and 
7r be an irreducible supercuspidal representation of G(k) constructed 
by J.-K. Yu and J.-L. Kim. In [HM08], Hakim and Murnaghan give 
a formula for the dimension of the space of such invariant linear func- 
tionals for all symmetric spaces. This formula reduces the computa- 
tion of the dimension formula to the "level zero part." For depth-zero 
supercuspidal representations, their formula reduces to the analogous 
problem over the residue field of k. This is one motivation to study 
such problems over finite fields. 

Over a finite field ¥ q of odd characteristic p, by convention through- 
out this paper, we use F to denote the Frobenius map F: G —> G, 
corresponding to the F 9 -structure of G, and we use the inner product 
(,)g f to denote the dimension of the G F -invariant Horn-space. Such 
dimension formulas for symmetric spaces have been studied by Lusztig 
[L90, LOO], Henderson [HO ] and others. In [L90], Lusztig gives a di- 
mensional formula for all Deligne-Lusztig virtual representations 

and for all symmetric subgroups H, that is, for (R^,lnd HF t)G F ■ Fur- 
ther, applying this formula for Rj,, Lusztig gives a dimension formula 
in the case when G F and H = G F in [ ], and Henderson [H03] 
solves the problem in the case when G = GL and H is any symmetric 
subgroup. 

The objective of this paper is to consider the case that the symplectic 
group G F = Sp 4n (F g ) and its symmetric subgroup H F = Sp 2n (F g 2). Let 
Jm be the skew-symmetric matrix given by 



where w n is the n-by-n permutation matrix with the unit anti-diagonal. 
Take G C GL 4n to be the group preserving the symplectic form defined 
by J 4n . Let 6 be the adjoint map Ad(e), where e is given by 
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and r is a primitive element in ¥ q . Since e 2 is in the center of the sym- 
plectic similitude group GSpf n , 6 is an involution of G F and the sym- 
metric subgroup H is isomorphic to ResF 92 /F g Sp 2n , i.e. H F = Sp 2n (F ? 2) 

where ¥ q 2 is the quadratic extension ¥ q (y/r) of ¥ q . 

First, we give an upper bound for the dimension of the subspace of 
Sp 2n (Fq2)-invariants in this case. In [Z10], we prove that the symmetric 
pair (G(k), H{k)) defined over non-archimedean field A; is a Gelfand 
pair, that is, 

dimHom G(fc) (7r,Ind2 ( ( J ) l) < 1 

for all irreducible smooth representations of G(k). Using the calculation 
in [Z10], we have 

Theorem 1.1. For each irreducible representation tt of Sp An (¥ q ) , 

Indeed, this can be deduced from H F gH F = H F g~ l H F for all g G 
G F over the finite field ¥ q and the calculation whose analogue over 
non-archimedean local fields was carried out [Z10]. It is easy to check 
that such a calculation holds for a finite field ¥ q of odd characteristic 
as well. We will omit the details in this paper. 

Our goal is to give a complete list of irreducible representations of 
G F which have a non-zero if F -invariant, which are simply called the 
distinguished representations. In this paper, we will focus on unipotent 
representations of G F , which occur as components of a Deligne-Lusztig 
virtual representation Rj,, where T is an F-stable maximal torus of 
G and 1 is its trivial character. For a general exposition, see Lusztig 
[L81, L84]. 

In this case, unipotent representations are parametrized by symbols 
of rank n and odd defect. In detail, if G is F-split, the F-stable maximal 
tori of G F can be parametrized by conjugacy classes of the Weyl group 
W of G, we use R^ w to denote a Deligne-Lusztig virtual representation. 
First, unipotent representations can be divided into families by 

R(c) = -±-tr(w,c)Rl w . (1) 

Here c is a cell defined by Lusztig, which is a virtual representation of 
W. In addition, R(c) is an actual representation of G F (not necessarily 
irreducible), with the smallest number of irreducible constituents one 
can expect from a linear combination of Rj> 's. In the GL ra cell c 

is just an irreducible module of the symmetric group S n and R(c) is an 
irreducible representation of G F . There is a one-to-one correspondence 
between irreducible unipotent representations of GL^ and irreducible 
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^-modules. However, this is not true for the other finite groups of Lie 
type. In the symplectic and odd orthogonal group cases, the irreducible 
modules of Weyl groups are parametrized by symbols of defect 1 as a 
subset of symbols of odd defect. 

Using Lusztig's formula in [L90], we have a dimension formula for 
Rt and then obtain a PU-module S such that for w G W 

tr(w,E) = (Rl w ,Ind%U). 

If G = GL, H is always an actual representation of S n and it is enough 
to decompose the module H into irreducibles. Once one has the multi- 
plicities of the irreducibles, the multiplicity of the corresponding unipo- 
tent representation of G follows. From the above discussion, this is not 
enough for the other reductive groups and creates difficulties in our 
case in comparison to the general linear group case. The issue is that 
even if we know the dimension formula for Rj,, there is still more work 
to be done in order to obtain the dimension formula for all irreducible 
unipotent representations. In addition to this issue, the IV-module S 
is usually not an actual representation in our case and it requires extra 
work here to decompose 5 into irreducibles. 

Define U(lnd HF t) to be the sub-representation of Ind^l whose ir- 
reducible sub- representations are unipotent representations of G F . Our 
purpose is to decompose the representation U(lnd HF t) into irreducible 
representations of G F . The strategy is to decompose the virtual repre- 
sentation E n of W2n, the Weyl group of Sp 4n (F g ), as 

n r 
r=0 /3hn— r i=l 

where pairs of partitions parametrize irreducible representations of 
W2n and (i, 2r — i) ■ (3 corresponds to an induced product character 
of S2r x S2n-2r- By the Littlewood-Richardson rule and combinato- 
rial arguments, we divide the irreducibles of S n into a family of cells 
c(Z, $ z , $z) such that 

(E n ,c(Z,$ z ,$ z )) =2 d 

where d is the number of singles in the special symbol Z. By the 
definition of cells, the associated representation R(c(Z, &z, &z)) of G F 
has exactly 2 d constituents. Since the multiplicity of each constituent 
is at most one, every constituent is distinguished in this cell. Therefore, 
all unipotent representations parametrized by symbols in c(Z, $2, <E>^) 
are distinguished and hence we have 
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/ (z, 2r - i) • A 



Theorem 1.2. 



w ( Ind sp::!? )!) = £iwr, <&*,**))■ 

In particular, the unipotent cuspidal representation of Sp 4n (F g ) /ias 
nontrivial Sp 2n (F g 2 ) -invariants. 

This result has applications to the theory of irreducible admissible 
representations of Sp 4n (/c) over a non-archimedean field and to the 
theory of automorphic representations of Sp 4ra over number fields. In 
Z13], we apply Theorem 1.2 and construct a family of distinguished 
representations of Sp 4n (fc). Further applications including the deter- 
mination of associated number-theoretic invariants such as L-functions 
and Arthur parameters and the construction of cuspidal automorphic 
forms which have a non-zero period integral associated to the symmet- 
ric pair will be considered in our forth-coming work. 

In this paper, we give an application of Theorem 1.2 to prove the 
rationality of unipotent representations of symplectic groups. This ex- 
tends the work of Lusztig [L02, Section 3] to all unipotent cuspidal 
representations for symplectic groups. We remark that our proof takes 
an elementary approach without using the Hasse principle. By Theo- 
rem 1.2, the unipotent cuspidal representation of G F is multiplicity-free 
in the Q[G F ]-module Ind^l. Since the characters of unipotent repre- 
sentations are Q-valued, the unipotent cuspidal representation of G F 
can be realized by a Q-module. Hence we obtain 

Corollary 1.3. All unipotent cuspidal representations of symplectic 
groups over finite fields of odd characteristic are defined over Q. 

The case of non-unipotent distinguished representations of G F is also 
of interest. This will be considered in a sequel to this paper. In that 
case we need to deal with unipotent representations of even orthogo- 
nal groups and general linear groups. Since the center of symplectic 
groups is disconnected, many arguments given here may be substan- 
tially changed. 

This paper is organized as follows. In Section 2, for the benefit of 
the reader, we recall Lusztig's classification theory of unipotent rep- 
resentations of symplectic groups (see Lusztig [L81] or Carter [ 93]), 
and Lusztig's general formula for in [L90]. In Section 3, we decom- 
pose the virtual W^n-module S 2n into well-understood W2n- m °dules. 
In Section 4, we give an explicit combinatorial description of unipotent 
representations in terms of Lusztig symbols. 
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Notation. Denote by S n the symmetric group of degree n. We also 
consider IV fi clS db longest Weyl element of S n . 

Let a be a partition of n with parts a\, 02, ... , a m . Its size, length, 
and transpose are denoted by \a\, £(a), and a'. The multiplicity of i 
as a part of a is denoted by mj(a) or m^. 

If G is a group and g £ G, denote by Clc(g) the conjugacy class of 
g in G and Zc{g) the centralizer of g in G. Let Z n be a cyclic group 
of order n. 



2. Unipotent Representations of the Symplectic Group 

In this section, we shall recall the Deligne-Lusztig representations 
in [DL76] and Lusztig's classification of irreducible representations of 
symplectic groups over finite fields. 

Let G be a connected reductive group defined over a finite field ¥ q 
of odd characteristic p and T be an F-stable maximal torus of G. Let 
A be a character of T F over Q* , where Qz is an algebraic closure of Q; 
and / is prime to q. For each pair (T, A), Deligne and Lusztig [DL76] 
attached a virtual representation R^ of G F . 

First, there is a bijection between geometric conjugacy classes (see 
Definition 5.5 in [DL76]) of the pairs (T, A) and semi-simple conju- 
gacy classes in G* F . Here G* is the dual group of G defined over ¥ q . 
Further, referring to [L77, Section 7.5], one has a bijection between 
G F -conjugacy classes (T, A) and G* F -conjugacy classes (T", s), with V 
an F-stable maximal torus in G* and s £ T' F . We also use to 
denote for the corresponding pair (T, A). 

Let $(G) be the set of isomorphism classes of irreducible represen- 
tations of G F over Q/. Let <§{G, s) be the subset of ^(G), consisting 
of all irreducible representations p such that (p, i?^) 7^ for some T. 
One has a partition 



<T(G) = J] <£((?, s) 



where s runs through the set of semi-simple G ! * jF -conjugacy classes in 
G* F . 

An irreducible representation p of G F is unipotent if {p,R^) GF ^ 
for some F-stable maximal torus T, that is, p is in <p(G, 1). In [L84], 
Lusztig parametrized all the unipotent representations of reductive 
groups. We refer to the parametrization of classical groups in Lusztig 
[L77]. 

2.1. Unipotent Representations of Symplectic Groups. In this 
section, we recall the classification of unipotent representations of sym- 
plectic groups and Lusztig symbols. See Lusztig [L81] or Carter [C93]. 
A symbol is an unordered pair A = (|!), where S and T are finite 
sets consisting of non-negative integers. In this paper, the symbol is 
reduced, i.e. ^ S fl T. The rank rk(A) of A is defined by 

rk(A) = £A + l>-[( #S + r _1 

xes tier IA 

where the square bracket is the greatest integer function. Define by 
the defect def(A) = \=ffS — #T|. Obviously, the rank of a symbol is a 
non-negative integer and rk(A) > [(def(A)/2) 2 ] . Lusztig used symbols 
to parametrize all unipotent representations of G F as follows. 

Proposition 2.1 ([L77, Theorem 8.2]). For the symplectic group, there 
exists a one-to-one correspondence between the unipotent representa- 
tions and the set of symbol classes of rank n and odd defect. More- 
over, the symbol A with rkA = [(def(A)/2) 2 ] corresponds to the cuspidal 
unipotent representation. 

For example, the unipotent cuspidal representation of Sp 2 ( d 2 +d - ) (F 9 ) 
corresponds to the symbol 

'0,1,2 2cf 



For each symbol A of rank n and odd defect we denote by p(A) the 
corresponding unipotent representation of G F . 
A symbol 

z Qi %2, ■ ■ ■ , Z2m 
^ z li z Zi " " ' > z 2m-l J 

of rank n and defect 1 is called special symbol if Zq < Z\ < z?, < 
• • • < Z2m-i < -22m holds. Assume that the number of singles in Z (the 
elements which occur only once) is 2d + 1. Let $ be an arrangement 
of the 2d + 1 singles in Z into d pairs and one isolated element. An 
arrangement is called admissible if it satisfies the following condition: 
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there is a pair consisting of consecutive z's and then the arrangement 
is still admissible for the new special symbol so-obtained by removing 
this pair. 

Let $ be an admissible arrangement of the special symbol Z . If \1/ 
is a subset of denote by \&* the set of z's in \l/ occurring in the first 
row of Z and the set of z's in \1/ occurring in the second row of Z. 
Let Zq be the set of z's in Z occurring twice. For any subset $ of $, 
define a virtual cell of W n , the Weyl group of Sp 2n (F g ), by 

where \l/ runs through over all subsets of $. This virtual cell c(Z, $, $) 
is considered as a virtual representation of W n . Indeed, each term in 
c(Z, $,$) is a symbol of rank n and defect 1 and not special except 

— 0. There is a one-to-one correspondence between the symbol 
classes of rank n and defect 1 and the irreducible representations of W n 
referring to Section 2.2. Those symbols of defect 1 are also considered 
as irreducible representations of W n . 

Similar to (1), we have a unipotent representation R(c(Z, $, $)) of 
G F 

R(c(Z,*,$)) = -^—Mw,c{Z^^))Rl w . 

| VV2n\ 

Referring to Lusztig's classification theory, we have 

R(c(Z,$,$))=J2p(^1 

i=l 

where Aj is uniquely determined by the virtual cell c(Z, $,$). More 
details may be found in Lusztig [L81]. 

Example 2.2. Let us give an example of unipotent representations of 
Sp 4 (F g ). All symbol classes of rank 2 and odd defect are 

1,2\ /0,2\ fO,l\ /0, 1, 2\ 1 , f /0, 1,2 



o y'V 1 / 'V 2 / 'V 1 ' 2 

The symbols in the first set are of defect 1 and also corresponding to 
all irreducible representations of Wi. The symbol in the second set is 
of defect 3 and corresponding to the unipotent cuspidal representation, 
that is, 9 W in Srinivasan's notation [S68]. Further, 



R[ andR 



0,1,2 
1,2 



are irreducible unipotent irreducible representations o/Sp 4 (F) and the 
first one is the trivial representation. For the special symbol Z = (°| 2 ) , 
$1 = {(0,1)} and $2 = {(1)2)} are the only two admissible arrange- 
ments and for instance 

i?(c(z,$ l! $ 1 )) = i?f°; 2 l -r(\ 2 ) =p( : 1 ) +010. (2) 



. 1 / V o , 

For additional complementary examples, see Carter [ )3, Chapter 13]. 

Example 2.3. Let G F = Sp 12 (F g ) and Z = (°^^) be a special symbol 
whose number of singles is 5. Then $ = {(0, 1), (2, 3)} is an admissible 
arrangement. We have 

ti((7**\\ f 2 ' 3 ' 4 V f°> 3 ' 4 V AU,2,3\ /0,1,2,3,4 
W$ ' $)) = V,1 J +P U,2 ) +P { 4 ) +P { 

(3) 

2.2. Irreducible Representations of the Weyl Group W n . In this 
section, we recall the construction of irreducible representations of W n 
in [L81, §2]. 

Let S n be the symmetric group of degree n acting on the set ,5f n = 
{1, 2, • • • , n}. Let a be a partition of n. Denote by ps(a) the irreducible 
module of 5" n whose restriction to S ai x S Q2 x • • • S ae{a) C S n contains the 
trivial character and whose restriction to S a > x S a > x • • • x S a ' C S n 

contains the Steinberg character. All irreducible modules of S n are 
parameterized by partitions of n in this way. For example, ps{n) is the 
trivial character and p(l n ) is the Steinberg character. 
Let W n act on the set 

X = {1,2,--- ,n,n',--- ,2',1'y, 

and be generated by + + for 1 < i < n — 1 and by (n,n'). 

Let V n be the set of all pairs of partitions (a; (5) such that \a \ + \(3\ = n. 
Then irreducible modules of W n are parametrized by the pairs in V n . 
Indeed, let (a; (3) be in V n , and ps(a) and ps{P) be the irreducible 
representations of S\ a \ and S\p\. Since W m is isomorphic to S m x (Z 2 ) m , 
there are natural liftings p~s( a ) an( i Ps(P) as representations of W\ a \ 
and W\/3\. Define a quadratic character \ °f W n via 

X ( W ) = (_l)#W(l)."'(2),-,«'(n)}n{l',2',-,n'}_ 

Then the induced representation 
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yj := Ind{^ o|XW|/J| p 5 (a) ® (p s (p) ® X\p\] 



is an irreducible representation of W n corresponding to the ordered pair 
(a; /3). 

By adding zeroes, we can increase the lengths of a and /3 at will and 
order < oti < a.- l+ i for 1 < i < m + 1 and < (3i < for 1 < i < m. 
By convention, at least one of a± and f3\ is not zero. Set Aj = a« + i — 1 
for 1 < i < m + 1 and /Xj = + i — 1 for 1 < i < m and then we obtain 
a symbol of rank n and defect 1 

\/3J ' \ ni,yv,'~ ,Hm ) ' 

Thus we have a one-to-one correspondence between irreducible repre- 
sentations of W n and symbols of rank n and defect 1. 

2.3. Lusztig's Formula for (tr(-, R^), Ind^l). For the virtual repre- 
sentations Rt, Lusztig gave a formula in [L90, Theorem 3.3] to compute 

tr(-, -R^), Ind^Fl^. We recall the formula in this section. Define 

9 T = {/ £ G | 6(f- l Tf) = r l Tf}. 

Then T (resp. H) acts on Qt by left (resp. right) multiplication. The 
double cosets T\Ot/H are in one-to-one correspondence with the dou- 
ble cosets B\G/H, where B is a Borel subgroup containing T. Let 
be the set of F- fixed elements. The double cosets T F \Q F /H F are also 
bijective to the G F -conjugacy classes of (9, F)-stable maximal tori of 
G F . 

For any / £ Q F , define a morphism €t,\ of (T fl fHf~ 1 ) F by 
e TX (t) = (_l) F 9- rank (^G((Tn////- 1 ) ))+F 9 -rank(z G (t)nz G ((Tn/^/- 1 )°)) > ^ 

Then €x,\ is a character and trivial on ((Tfl Define 

@t,a = {/ e @r I X {TnfHf~ 1 ) F — ct,a}- 

The groups T F and i? F still act on A by left and right multiplication. 

Proposition 2.4 ([L90, Theorem 3.3]). 

(tr(.,i$),Ind£l) = ^ ( _ 1) F 9 -mn fe (T)+F 9 - ra „ fc (^((Tn/^/-i)°)) 
feTF\e^ x /HF 

Remark that this formula holds for all connected reductive groups 
and its symmetric subgroups. 
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3. Decompositions of W^-module H 



3.1. Double Cosets T\Q/H. Let G be the symplectic group Sp 4n . 
Fix T to be the set of diagonal matrices of G, and B to be the set of 
upper triangular matrices of G. Then B is an F-stable Borel subgroup 
of G containing T. Let g be an element of G such that g~ 1 F(g) G 
Nq(T). Then gTg~ x is also an F-stable maximal torus of G. As G 
is F-split, the map from gTg^ 1 to the image of g~ l F(g) in W gives a 
bijection between the G F -conjugacy classes of F-stable maximal tori 
of G and the conjugacy classes of W. Let g w be an element in G such 
g^FiQw) = w. Denote by T w = g^Tg" 1 the corresponding F-stable 
maximal torus. 

Let •§ be the map on G defined by $(g) = g9{g~ 1 ). Then •§ gives a 
bijection from G/H to the image ^(d) of d. Note that 



Recall that T is a ^-stable maximal torus of G. Let N(T) be the 
normalizer subgroup of T in G and M = N(T) D ^(i?). Define a 9- 
twisted action of T on Af via t * u> = tuo6(i)~ l . This twisted action 
also induces an action of T on N(T)/T, which is the e-conjugation. 
According to [HW93, Proposition 6.8], G is the disjoint union of the 
double cosets BjH, where T * $(7) runs through all the obits of the 
action of T on M . 

In our case, the Weyl group N(T)/T is W 2n - For x G N(T), de- 
compose x as x = tve' where t G T and t> G H^n (consider t> as a 
representative in N(T)) and 



^)c{ 9 GG|%)=^}. 



/ 



n 



\ 



e' = 



n 



e Sp 4n (F 9 ). 



T 



n 




(5) 



J^ (1) = {i G J^ 2n I «(z) G e^ 2n ,t;(z) > t} , 

j?W = {iej? 2n \v(i)=i}, 
y^ = {iey 2n \v(i) = i'}. 
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Note that the sets y^\ v{y v (1 >), y v [Z> , v{y v (Z >)', and y v w are a 

partition of y^n-, where i" = i. Then, v is of form 



.(i) 



(2) 



»(2)y 



.(3) 



(i) 



(4) 



Denote by a„, fe^ and c„ the corresponding product over 

for f = 

1,2,4, respectively. Note that each factor commutes with the others. 
We choose the following representatives in G F corresponding to the 
cycles (i,j)(i',f) and (i,j)(i',j')(i,i')(jJ') respectively by embedding 
the 4x4 matrices 

/ 1 \ / 1 \ 



1 



V 



1 



and 



/ 



-1 



V 



/ 



into the 4n x 4n matrices in the (i, j, f, i')-th rows and (i,j,j',i')-th 
columns, with 1 for the remaining diagonal entries, and zeros for all 
the remaining non-diagonal entries. 

Denote by Q the set of all pairs (t v ,v) in T x W satisfying the 
following conditions: 



(4) 



(tv~)i 




(3) 



\y?\ where 



m v = #{* G y^ I (t v )i = 1/v^}. (6) 

Let "f(t v ,v) be an element in G F such that i?(7(£„, v)) = t v ve'. In- 
deed, there exists an element g\ in GL 4n such that g\sgf l = t v ve'e, 
since both e and t v ve'e are GL 4n -conjugate to diagjy^-^n, — V^2n}- 
Since e G GSp 4ra , there exists an element 7(^,1;) in Sp 4ri such that 
j(t v , v )ej(t v , v )~ 1 = t v ve'e and then <d{j{t Vl v)) = t v ve' . In addition, 
{t v ve' I (t v ,v) G fi} is a subset of Af. 

Lemma 3.1. The set of double cosets B\G/H is in one-to-one corre- 
spondence with the set Q. Moreover, {j(t v ,v) \ (t v ,v) G f2} is a set of 
double coset representatives. 

Proof. Let x = tve'. Since the (i, i)-th entry of 9(x)x is — 1 for % G 

9(x) 7^ x^ 1 and x is not in Af if y^ ^ 0. Thus for x G Af we may 

assume x = ta v b v e' for some t = diag {ti, t 2 , . . . , t 2 n, t2.ni • • • > ^2 ^i" 1 } e 
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T. Let #W for 1 < % < 3 be elements in T, and ^- be the (j, j)-th entry 
given by 

and 

' \i jey^xiyp'u^)'). 

Then t = t^H^H^. Let d T = diag{r/ 2n , t- 1 /^}. We also have the 
decomposition d^d^d^ corresponding to t^. Further, 

tvtv- 1 = ta v ta- l b v tb- 1 = a^^a" 1 ^ 2 ^ 1 ^) 2 , 

and 

diag {rl 2n , I 2n } wdmg {l 2n , r^hn] w = d^ ] a 2 w b 2 w d^ ] = d^d®. 
By 6{x) = x~ l and Equation (5), we have 

hn =tv fo _1 diag {r/ 2 „, I 2n } v diag {l 2n , t' 1 ^} v 
=4 1 ¥ 1 W (1 V ■ t^Kt^b- 1 ■ (t^) 2 d?\ 

Thus 

Urt v(l) )- 1 iey v {l) 
U = I t v( i Y % e y v {2) (7) 
[±l/v^ «ey (3) . 

Denote by m+ = #{z G ^ (3) | U = l/JF) and = #{* € 
t 5^i 3 ' ) | U = —1/y/r}. Since x G A/", there exists g <E G such that 
90(g) 1 — x an d then geg~ x = tve'e. The eigenvalues of e are ±V^" 
and the dimension of the eigenspaces is 2n. We have 

tve'e = ^W (2) M (3) diag {r/ 2n , J 2n } , 

and decompose diag {rl 2n , J 2n } as (e'e)( 1 )(e / e)W(e / e)( 3 ) similarly to the 
decomposition t = t^H^H^. 

By Equation (7), the dimensions of the ±A/r-eigenspaces correspond- 
ing to the matrix t^a v (e'e)^ are 2# t 5*'J 1 \ the dimensions of the ±\/t- 
eigenspaces corresponding to the matrix t^b v (e'e)^ are 2#^ , and 
the dimensions of the i-y/r-eigenspaces for the matrix v^>(e'e)^> are 
2m + and 2m~. Thus m + = m~ = |j?« 3 ^. 

Let h G T '. The 0-twisted action /i * x is given by 

hx6{h)- 1 = /i( 1 )t( 1 )(a 1 ,(/ i ( 1 ))- 1 a; 1 )(/ i ( 2 ))-¥ 2 )(c>( 2 ) C ; 1 )^'. (8) 
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Choose the element fteT defined by 



/?.; 



l ey 2n \(yi 1) uyi 2) ). 



By Equation (7) and (11), h * x = t v ve'. Each T-orbit on Af has 
a nontrivial intersection with {t v ve' \ (t v ,v) G Q}. In addition, by 
the above discussion, for the different pairs (t vi ,vx) and (t V2 ,v 2 ), the 
elements t vi vie ! and t V2 V2s' are in different T-orbits. This completes 
the proof of the lemma. □ 

Corollary 3.2. The set {y(t v ,v) \ (t v ,v) <E Q} is a set of coset repre- 
sentatives for T\Q/H. Furthermore, Q w := {g w ^{t v ,v) \ (t v ,v) G fl} 
is a set of coset representatives for T w \Q Tw / H . 

Since T w and H are connected, and 9 and F commute, the action of 
F on G induces an action on the double cosets T w \Q Tw /H. In addition, 
since the group T w fl ■yH r y~ 1 are connected for all 7 in T w \Q Tw /H , the 
action of F on T w \®t w / H is a permutation on the representatives Vt w . 
Let Q F be the subset consisting of all fixed orbits under the action of 
F. Then 

t^\Q f t Jh f » n F w . (9) 

Define 

fi± = {(*«,«) G ft I v G and w^w" 1 = -(t v )i for i G ^, (3) } . 

Lemma 3.3. If T w is an F -stable maximal torus corresponding to w, 
then Q F = Q±, i.e. there is a bijection 

t*\® f t Jh f » ni. 

Proof. F(g w r y(t v ,v)) is in the double coset T w ■ g w -f(t v i,v') ■ H if and 
only if for some t G T 

wF{t v )ve' ew~ l = tt v ve 'et~ 

<=^ wF(t v )di&g {r J 2n , hnY w~ l ■ wvw' 1 = t„diag {r/ 2n , hnY • tvt' 1 

where diag {r^n, hnY = v ' diag {r/ 2r) ,, J 2n } • v" 1 . Then v commutes 
with w in W. Since v 2 = I± n and v G Zw(w), wvw' 1 is T F -conjugate 
to v and tvt' 1 ■ wv~ 1 w~ 1 = to for some to m the connected component 
of the f-split torus {t G T \ vtv^ 1 = t^ 1 }. The statement is equivalent 
to 

wF(t v )di&g {r/ 2 „, I 2n } v w~ l = t t ,diag {r/ 2n , I 2n } v ■ t 

for some to in the connected component of v- split torus. Let g = 
t„diag {r/ 2n , J 2 „} 1 '. It is enough to prove that wF(g)w~ 1 g~ 1 is in the 
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connected component of f-split torus. Since vgv 1 g = rl^ ni the exis- 
tence of to is equivalent to w^^itv^ 1 = — {t v )i for each i G under 
the assumption v G Z\y{w). Then this lemma follows. □ 

According to [< 93, Theorem 3.5.6], for a connected reductive group 
G, if its derived group is simply connected, then Zq{s) is connected for 
any semi-simple element s. Since G F is the symplectic group Sp 4n (F g ), 
Zait) and T n fHf' 1 in the definition (4) are connected. Thus e^j is 
trivial. If the character A is trivial, then T F \Q F X /H F is the same as 

t f \q f /h f . 

Example 3.4. Let G be Sp 4 . By the lemmas above, it is easy to check 
that Q contains 4 elements. If (t v , v) is in Q, then v is the identity e or 
(1,2)(1',2 / ) or (1, 2')(1', 2). Ifv = e, there are two choices oft v , which 
are ±di&g{l/y/r, — 1/y/r, — y/r, y/r}. If v ^ e, t v is trivial. Hence we 
simply use {(1, 2), (1, 2'), (+, — ), (— , +)} to parametrize these 4 double 
cosets. We give the centralizer Z G (T w n fH f~ l ) for each F -stable torus 
T w and the associated double cosets. 





(1-0) 


(2;0) 




F x x F* 




(1,2) 


GL 2 (F 9 ) 


GL 2 (F g ) 


(+,-) 




F x 2 

q 2 


(-,+) 






(1,2') 


GL 2 (F 9 ) 


U 2 (l» 




2 


2 



(i;i) 



(0;2) (0;T 



FT 



x n> 

U 2 (J,F, 2 ) 



U 2 (J,F 9 2 
-2 



The first row lists the pairs of partitions in Vi parametrizing the 
G F -conjugacy classes of F-stable maximal tori. The last row is the 
dimension formula < i?^,Ind^Fl >• The symbol '—' means that the 
corresponding double coset is not F-stable and not in the set 

3.2. Decomposition of H n . There is a one-to-one correspondence be- 
tween pairs of partitions (a;/3) such that \a\ + \(3\ = n and the con- 
jugacy classes in W n . If i and j are parts of a and (3 respectively, 
we map % and j to Coxter elements in Si and Wj respectively, which 
extends to a bijection between pairs of partitions (a; (3) in V n and the 
conjugacy classes in W n . Denote by Clw,, + . . (a; (3) the conjugacy class 
corresponding to (a; (3). If w is in Win of cycle-type (a; (3), then 

T£ = IIK" 1 ^ x H(K») mm and *Vrank(Tj = %), 

i i 

where F* 2l is the unitary group Ui(F ? 2i). 
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Let (a- (3) be in V 2n - Let h{a) = {(1 l'),(2 2'),. ••,(%) %)'))} 
where (i i') is a permutation in W^n, and h{(3) = {1,2, . . . , £(/3)} be 
sets indexing the parts of a and (3 respectively. Define 

P 1 (a) = {((i | (i i') G h(a) and j G Z a .} 

and 

P 2 (/3) = {{hj) I i G / 2 (/3) and j G Z 2ft }. 
Then w is a permutation of Pi (a) and P2(/3) by w(i,j) = + 1). 

For each (t v ,v) G f induces permutations of Pi(a) and of 
order 2, denoted by v and -u respectively. There are unique elements 
ji(v,i) and j 2 (v,i) such that = {v(i),j + ji(v,i)) for G 

Pi(ai) and v(i,j) = (v(i),j + 32(^,1)) for G P2(/?)- Note that 

for (i i') and (j j') in ii(a), u(i i') = (j j') and i') = (f j) are 
considered as two different permutations of {(i i'), (j j')}. 

Define 

t v (a) = #{(i i') = (i' 0} 

and 

^(«) = #{(* i') G h(a) I u(i i') = (i i') and ^(u,*) = 0}. 
Also define 

IM = #{* G J 2 (/3) I C(i) = j and 2 ^ j}. 

Note that £„(/?) is always even and fi™ 7^ if and only if rrik(a) and 
rrik(P) are even for each odd part A; of a and (3. 

Define a W^-module S n such that the character of S„ is given by 

tr(w, S n ) = (tr(-, Ind^l) . (10) 

Lemma 3.5. 

tr(w,~ n )= J2 (-1)^)+^)+^)+^ 
(t v ,v)en% 

where (a; (3) is the pair of partitions associated to w. 

Proof. We decompose the symplectic space V = (F g ) 4n according to 
the invariant subspaces of T F 

V = ® { i, v)eh{ a){Vi © Vi.) © ®iei 2 ^)Vi, 
where F*, 4 for (i 1') G h{a) and T F \ Vi 2i F 1 ^ for i G 

12(0)- Fix an element (t v ,v) G fi± and denote by 7 the corresponding 
representative of the double cosets T F \Q F / H F . 

Let (i i') be an index in I±(a) and assume v(i i') = (j j') or (j' j). If 
2 7^ j, then the restriction of Z G (T W f\^H^~ l ) into the invariant space 
Vi © Vj or Vi © Vj/ is isomorphic to GL 2 (V r i © 1^) and its F^-rank is 2. 
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If i = j, then aij is even due to the existence of t v and we have two 
cases, ji(v,i) 7^ and ji(v,i) = 0. If v(i i') = (i i') and ji(v,i) = 0, 
then the restriction of ZG(T w r\jH'j~ 1 ) into the invariant space Vi © Vy 
is the same as T^l^^, = F* Qj and its Fg-rank is 1. If v(i i') = (i i') 
and ji(v,i) ^ 0, the restriction of Zq(T w fl •yHj' 1 ) is isomorphic to 
GL 2 (Vi) but invariant under and its F g -rank is 2. If v(i, i') = (i f , i), 
the restriction of Zq(T w H^H^ -1 ) is isomorphic to U 2 (Vi © Vy) and its 
F 9 -rank is 1. 

Let i be an index in I 2 {fa) and v(i,i') = (j,f) or U i ^ j, 

the restriction of Zq(T w D jH'j -1 ) into the invariant space © Vj © 
V}/ © Vi' is isomorphic to U 2 (Vi © Vj) and its F g -rank is 1. If i = j, then 
j 2 (fa, i) = and fa is even due to the existence of t v . The restriction of 
Zq(T w fl ■yH r y~ 1 ) is the same as T F |v; = F^. and its Fg-rank is 0. 

In sum, the F g -rank of Zg(T w fl ^H^ 1 ) and ^(a) + i£(a) + 
have the same parity. By Proposition 2.4, this lemma follows. □ 

Recall that W 2n — S 2n ^ (Z 2 ) 2n acts on £? 2n . Take the subgroup, 
isomorphic to S n x S n , of W 2n consisting of all permutations on the 
{1,2, ... ,n} x {n + 1, . . . , 2n}. Let a n = YYi=i(h 2n + 1 — z). Then a n 
in normalizes 5„ x S n , that is, cr n is the longest Weyl element w 2n . 
Let K n = ((<7 n ) x x S' n )) x (Z 2 ) 2n regarded as a subgroup of W 2n 
and sgn K be a character of lifting from the non-trivial character of 
the group {e,a n }. Define a virtual module of W 2n 

Kn = Ind^"l-Ind^"sgn K . 

Let N n be the centralizer Z W2n (a n ). Then N n is isomorphic to (S 2 I 
S n ) x (Z 2 ) n where S 2 I S n is the wreath product. Define a character 
sgn^ of N n by 

Sgll^/i) = (_l)#{^(l)^(2).-,ft(n)}n{l',2',...,n'}_ 

Let v n = Ind^sgn^. 

A function / on the set V 2n is called multiplicative if f(a; fa) = 
Yli f(i mi ( a '\ 0)/(0; i m ^) where (i mi ) is a partition of i ■ rrii. Now, we 
decompose n n and v n into irreducible representations of W 2n . 

Lemma 3.6. 

Moreover, the character function tr(-,u n ) is multiplicative and if w is 
an element of cycle-type (a; fa) then the following holds: 

(1) ti{w,v n ) = when w is of cycle-type (i m ;0) or (0;i m ), and i 
and m are odd; 
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(2) tr(w, u n ) = i m ^^ when w is of cycle-type (i 2m ; 0) and i is odd; 

(3) tr(w,u n ) = (—i) m ^r~ when w is of cycle-type (0;i 2m ) and i is 
odd; 

(4) tr(w,u n ) = when w is of cycle-type ((2i) m ;0) or (0;(2i) m ), 
and m is odd; 

(5) tr(u>,i/„) = (2i)T ^2)\ w h en w ^ s of cycle-type ((2i) m ;0) and 
m is even; 

(6) tr(w,u n ) = (— 2z)t jz^nm when w is of cycle-type (0; (2i) m ) and 
m is even. 

Proof. First, we give a set of representatives for the double cosets 
(W ni x W n2 )\W 2n /N n where ni < n 2 and n x + n 2 = 2n, which is 
the same as S ni x S n2 \S2n/ $2 I S n . Let 



Km = Y[(m + 1 - h 2n + 1 - i) for < r < (11) 

i=l 



Then {/i rjni | < r < [|]} is a set of representatives for the double 
cosets S ni x S n2 \S 2n / S 2 I S n . Denote by h rm the embedded element in 
W 2n and {h r , ni | < r < [|]} is a set of representatives for the double 
cosets W ni x W n2 \W 2n /N n . 

Let (°) be the irreducible module of W 2n given in Section 2.2. If 
\a\ 7^ let = min{|a|, Then (7 = {n,n'){n + 1, (n + 1)') 

commutes with h rjU1 for all < r < [^]. Since g G iV n and (7 = 

KmgKX G ^i«i x w \p\> we have s s n iv(^) = - 1 and x\ts\(g) = l if 

|ot| < or ps(a)(g) = 1 if |a| > By Mackey's theorem, 
(f o h^Wan = when |a| ^ \/3\. 



Now, we only need consider the case \a\ = \/3\ = n. If r 7^ 0, then 
Kngh'n = (2n + l,(2n + l)')(n + l,(n + l)') is in W\ PV Also sgn N {g) = 
-1 and x\p\{hr, n gh~)i) = 1- Denote by N' n = (W n x W n ) PI iV n . This is 
isomorphic to 5„ x (Z 2 ) n where S 1 ^ is the subgroup of S n x S^, and 
commutes with a n . Using Mackey's theorem and Frobenius reciprocity, 
we have 

a \vn)w2 n = (ps(<*)® Ps{P),l)s£ = (ps(<*),Ps(P))sn- 

Next, let us calculate tr(w, u n ) where w is an element of cycle-type 
(a; (3). We discuss the values at w in each case separately. If m^a) or 
rrii(/3) is odd for some odd i, then Clw 2n (w) HiV n = and tr(w, u n ) = 0. 
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If w is of cycle-type (i 2m ; 0) or (0; i 2m ) and % is odd, then Clw 2n ( w ) n 
N n contains at most one iV n -conjugacy class in N n . If w is in N n , then 

tr(w v ) = I^MI = i 2m ^m)\1 2m = (2m)! 
1 ' n) \Z Nn (w)\ i m m\2 m 2 m ml 

when w is of cycle-type {i 2m ; 0) and 

when w is of cycle-type (0; ? 2m ). 

Let u> be an element of cycle-type (0; (2i) m ). If m is odd, then 
C/^ 2n (w) fl N n — 0. If m is even, then C/vi/ 2n (u') H iV n contains at most 
one iV n -conjugacy class in N n . If w is in iV n , then 

mCZw^u;) , .m (4i) m m! (-2i)?m! 
tr(tt '"" ) = 1 WM = 1 (2i)Tfl2T-2T = 

Let u> be an element of cycle-type ((2i) m ; 0) and Clw 2n ( w ) ni\r n = 0. 
In this case, Clw 2 „( w ) ^ contains 2[y] + 1 conjugacy classes of iV n . 
By the formula for the induced character, 



Z K .M (2i)?f!2?-2? k ' (m/2)! 

when m is even and tr(w, z/„) = when m is odd. In addition, it is easy 
to check that tr(-, K n ) is multiplicative. This completes the proof. □ 



Lemma 3.7. 



<.=B-i>f V) 



i=0 

Moreover, the character tr(-, /c n ) zs multiplicative and ifw is an element 
of cycle-type (a; (3) then 

{ (a; P) = ((2i + l) m ; 0) or (0; (2< + l) m ) 
1 ' W \2 m (a;P) = ((2t) m ;0)or(0;(2i) m ). 1 J 

Proof. Since the characters 1 and sgn^ of K n are trivial on the sub- 
group (Z 2 ) 2n of W 2n , it is sufficient to show that 

n 

Ind^l - Indfrsgn^, = ^(-l)Vs(i, 2ra - i), (13) 

i=0 

where ^ = (cr n ) x (SVt x 5 n ) is the subgroup K n fl S 2 n and sgn^/ is 
the restriction of sgn^ on K' n . 
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First, applying Pieri's formula, we have 

n 

Ind£" x5n l = Ind^l + Ind^sgn^ = ]T p s (i, 2n-i). (14) 

i=0 

Let g r = YYi=i(h 2n+l—i) be an element in S 2n (g r — e when r = 0, and 
9n = a n)- Then g r for < r < n are a complete set of representatives 
for the double cosets (S n x S n )\S 2n / (S n x S n ) and S^n = lll^o 1 K n9rK' n . 
Since g r and <r n commute for all r, cr n is in K^HgrK^g' 1 for each double 
coset K' n g T K' n in S 2 n- Since l(cr n ) 7^ sgn K ,(cr n ), by Mackey's theorem, 

Indf, n l,Ind*,"sgn x ,\ =0 (15) 

/ s 2n 

and 

'ind&l.^l) = *K\S2n/K = [£] + 1. 



Referring to Macdonald [M95, VII (2.4)] 

^ S 2 n 



«r xSn l,Ind| k l> & = £]+!. 



Also we have 

(indffl.Indgfc n l) = #K' n \S 2n /S 2 I S n = Q] + 1. 
Hence Ind^l is a submodule of Indf^l. By (14), 

[n/2] 

Indf^l = ^ p 5 (2i, 2n - 2z). 

i=Q 

Then Equation (13) follows by (15). 

Next, let us calculate tr(w, K n ) where w is an element of cycle-type 
(a; (3). If a part i of a or /3 is odd, then Clciw) D = CIkS w ) ls * n 
the subgroup x (Z 2 ) 2n . By the formula for the induced character, 
tr(w, /c n ) = 0. 

If Clw 2r ( w ) an< i the left coset a n ■ K' n x (Z 2 ) 2n have a non-empty 
intersection, then CIq(w) C\K n is the disjoint union of at most two K n - 
conjugacy classes. If there are two conjugacy classes, one conjugacy 
class is in K' n x (Z 2 ) 2n and the other is in a n ■ K' n x (Z 2 ) 2n . In all cases, 
we have 

_ 2\W 2n \ \Cl W2n (w) n a n ■ K' n x (Z 2 ) 2 "| _ 2Z W2n (w) 
tI[w ^ n) ~ \K n \ ' |CV 2 »| " ' 

Hence, tr(w, K n ) is multiplicative and Equation (12) follows since 

Z W2n {w) = (2i) m m!2 m and Z Kn {w) = 2 • i m m\2 m 
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n 

*-"n 



EEE<-i)'r V ) (16) 



when (a; 13) = ((2i) m ;0) and 

ZwM = (4*) m m! and Z Kn {w) = 2 • (2i) m m\ 
when (a; (5) = (0; (2i) m ). This completes the proof. □ 
Proposition 3.8. 

(i, 2r - i) 

r=0 /3hn— r i=l 

where (P) • (l 2r_;i ) . /3 corresponds to the representation 

Ind S s ;% Sn _ r ps( i ,2r- i )®p s (P). 
Proof. It is equivalent to show the following identity 

n 

Zn = J2 lnd w7r><W 2(n ^r ® I/„-r- (17) 
r=0 

We will match the characters of the two sides. 

Let w be an element in W 2n of cycle-type (a; (5). The set Q?± may be 
written as a disjoint union Ur=o^±,r> where 

n^ r = n%n{ve w 2n \ jr® = 2r}. 

Define the set 

V r (a; p) = {(a, b) E V 2r x V 2n . 2r \ a U b = (a; 0)} 

where the union a U b is the component-wise union. We may further 
decompose fi™ as 

^Ir = II <r X fig,. 

(a,6)eP r (a;/3) 

where u;' and iu" are of cycle-types a and 6 in W^r and W 2n - 2r respec- 
tively. Note that if (t v ',v') is in Q± r , then v' is the identity and we take 
t' G Q^y, if (V', u") is in fig, then V = 1 and we take v" € Q|" . It is 
easy to observe that £(a) = ^(a) = £(o/), ^(a) = V'(a"), 

and l v {(3) = £{P") where a = {a'; /?') and b = {a"; (3"). Then 

n 

r=0 {t v ,v)en^ r (a,b)eV r (a;{3) 

( ^ (_1)2^«'))( ^ (-l) / ( a ")+V'(a")+<(/8")/2)_ 
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We consider the characters tx(w',E' r ) and tv(w", H"_ r ) on Wir and 
W 2n -2r defined by tr(w',E.' r ) := #fi±' r and 

By the definition, the class functions tr(«/, Sj.) and tr(w", H^_ r ) are 
multiplicative. In order to prove (17), it is enough to show that the 
characters tr(w',E' r ) and ti(w" ,E'^_ r ) match tr(-,K 2r ) and tr(-, i/ 2n _ 2r ) 
on the conjugacy classes of cycle-types (i m ; 0) and (0; i m ). This is easily 
verified by Lemma 3.7 and Lemma 3.6 and the definition of tr 
and tr(-, S"_ r ). Then this lemma follows. □ 



4. Distinguished Symbols 



In this last section, we will use the Littlewood-Richardson rule to 
decompose S n into irreducible representations of Win and then divide 
those constituents into a sum of virtual cells. Doing so, we conclude 
that there is a bijection between symbols in those cells and the distin- 
guished unipotent representations. 

The set-theoretic difference of the Young diagrams of a pair of par- 
titions (a; 0) is called a skew diagram of shape a//3 and size \a/f3\ 
is | a | — \(3\. Let (a//3)i = on — A- A skew diagram is a horizontal 
strip (resp. vertical strip) if it contains at most one box in each column 
(resp. row). For a skew diagram 77, let h(rf) be the horizontal strip 
obtained by removing all columns from rj which contain more than one 
box and 11(77) be the skew diagram obtained by removing h(rj) from 77. 
For example, 



V 



□ 



□ v (77) 



Denote by I /i (77) I (resp. |v(?7)|) the number of boxes of h (77) (resp. v(rj)). 

A horizontal strip 77 is called even if the number of boxes in each row 
of rj is even. Let T 2 be the set of all skew diagrams which contain at 
most 2 boxes in each column and is of even size, and T% be a subset of 
T 2 of all skew diagrams 77 such that h(r)) is even. 

Let 77 be in T 2 . Define a tableau T of shape 77 by putting the integers 
'1' or '2' in each box of r\ such that the follow Littlewood-Richardson 
condition holds: The integers in all boxes are listed from right to left 
then from top to bottom, then at the first k entries in this list for each 
1 < k < \r)\, each '1' occurs at least as many times as '2'. 
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Define Tab(r^) to be the set of all tableaus of shape 77 and Tab(?7)j to 
be the subset of Tab(r^) consisting of tableaus of i '2's. For example, 
when z > iy, Tab(?7)j = and | Tab (77)^ = 0. By convention, if = 0, 
define Tab(?7) = {0} and |Tab(r/)| = 1, where |Tab(ry)| (resp. |Tab(^)i|) 
is the size of the set Tab(r^) (resp. Tab(?7)i). Also we have the disjoint 
union Tab(r^) = U.;>oTab(?7)j. 

Next, we use the Littlewood- Richardson rule to decompose H n into 
irreducibles and get the following lemma. 



Lemma 4.1. 



£ (-1: 



2 



«//3er° 



P 



Proof. By Equation (16), if (E n , y2)) 7^ 0, then (3 < a and a//3 G T 2 . 
If a = {3, then a//3 is in and (H n , (^)) = 1. 
If /3 < a, by the Littlewood-Richardson rule, 

\a/fS\/2 \ a /p\/2 

[qP = E ("iriTabK/?),! = Yl ("iriTabW/?),!. 

i=0 i=\v{a/0)\/2 

Since |Tab(a/ / 5) l -| = |Tab(/i(o;/^)) i _|t,( a / / 3)|/ 2 |, we have 

|h(a/jB)|/2 

(s B ,Q> = (-i) |w(a/ » |/a E {-\m^{h{a/m- 

For a tableau T in Tab(r/), let |T 2 | be the number of boxes filled with 
2's. For a subset T of Tab (77), define 

cm = E(- 1 ) |r2 '- ( 18 ) 

TeT 

Next, we will show that for a horizontal strip 77 of even size 

«,)) = ft ,f t r even; (w) 

I otherwise. 

Let T(?7)(i) be the subset of Tab(7/) consisting of all tableaux whose first 
box filled with £ 2' occurs at the right-end box of the i-th row. By the 
Littlewood-Richardson condition, T{r]){\) = 0. Denote by T{rj)^ + i 
the subset consisting of the one tableau whose boxes all contain '1', 
where £(r)) is the number of rows of 77. We have a disjoint union of 
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Tab(?7) and a formula of ( (Tab (77)): 



%)+l e{rj) 

Tab(ry) = ]J T(77) W and ((Tab^)) = 1 + £ C(Tfo) W ). 

i=2 i=2 

In order to prove (19), we only need to consider the rows whose numbers 
of boxes are non-zero. 

In order to evaluate CO'iv)^)) f° r 2 > 2, we reduce to a skew diagram 
77 w of a smaller size than 77. Let 77 W be the horizontal strip such that 
the numbers of rows are (X)}=i — 1 , ^7* — 1 , . . . ,?]t(rj)) from top to 
bottom, obtained by merging the boxes in the top i — \ rows of 77 to the 
7-th row, becoming the first row of X^=i Vj boxes, and then removing 
the right end boxes in the top two rows. For instance, 



12 



v 
1 1 
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Then we obtain a bijection between T{jj)u) and Tab^w), and 

C(T(vW = -C(Tab(77«)). 

Now, we have 

t( v ) 

C(Tab(r/)) = l-^C(Tab(77W)) (20) 

i=2 

and 1 77 j = |?y| — 2 for 2 < i < i(r)). We apply this inductive for- 
mula (20) to prove (19). If \r)\ = 0, C(Tab(77)) = 1. If \rj\ = 2, we 
have two types of skew shapes, I I and B, denoted by (2)/(0) and 

(2, 1)/(1) respectively. Then it is easy to check that £(Tab((2)/(0))) = 
1 and C(Tab((2, 1)/(1))) = 0. 

In general, if 77 is even, then 77 W is not even for all 2 < i < £(77) and 
[77 w ] < \rj\. By induction, ((Tab(r]^)) = and then £(Tab(77)) = 1 
by (20). If instead 77 is not even, let 2 max be the maximal integer 
such that 77j max is odd. Then 77W is even if and only if i = 7 max . By 
induction, we have C(Tab(7/ i )) = 1 when i = 7 max and C(Tab(77*)) = 
when i ^ i max . Hence C(Tab(77)) = 1 - C(Tab(7/( w ))) = 0. This 
completes the proof of (19) and the lemma follows. 

□ 
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For example, when G = Sp 4 , Si = ps(2) — ps(l 2 ) + Q). If G — Sp 
then 
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-h:;h:kmo-(?) +(';■' 

J) - (3 * - (?) * Cf 

Let <S n be a set consisting of all special symbols of rank 2n whose 
associated pairs of partitions under the map L~ x are even horizontal 
strips, that is, £~ 1 (5 n ) is the same as the subset of Y° 2 consisting of all 
horizontal strips of size 2n. Given a special symbol 

Ai, A2, • • • , A m+ i 

define an admissible arrangement 

®z = {(K,fJ>i) I \ ^ Hi, for 1 < i < m} 
and a subset of <J>z 

§z = {(Ai, e $ I ^ - fii = 1 mod 2} . 
Theorem 4.2. 

In particular, the unipotent cuspidal representation of Sp 4n (Fq) has 
non-trivial Sp 2ri (F g 2 ) - invariants . 

Proof. First, we prove that 

~ n = ^c(Z,f,$ z ). (21) 

Assume that a//5 is in T° and |e*| + \0\ = In. Then 1(a) - £(/3) < 2. 

Recall that we increase the lengths of a and j3 such that = 
£(f3) + 1 by adding zeros. Assume that a = (a±, a 2 , ■ ■ ■ , cxm+i) an d 
(3 = (fa, (32, ... , (3 m ) with «i < ccj+i and fa < fa + \ and at least one of 
cti and /?i is nonzero. 

Since (3 < a, we have < atj+i for 1 < i < m. Set Xi = on + i — 1 
for 1 < 2 < m + 1 and /i« = + i — 1 for 1 < % < m. Then the symbol 
A = ( A ) is of rank 2n and defect 1. Since /3 < a, A is special if and 
only if ^ < fa for 1 < i < m. Then the skew diagram a/ (3 has no 
two boxes in each column, that is, a/ (3 is a horizontal strip and even. 
Hence, A is special if and only if A is in S n . 
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AC*) 



Let \I/ be a subset of $z. For each subset \& C define 

^oLJ^LI(^-^)* N 

ZoTJ**II($-*), 

and denote by C~ 1 (A{^f)) = (^,). Since max{ai,/3j} < minjaj+i, Pi+i} 
for 1 < 2 < m (when i = m, let min{a i+ i, Pi+\} = a m+ i), we have 
/?i < for 1 < i < m. 

We will show that for all subsets \1/ of 



-l)l* n * z l£- 1 (A(^)) 



1. 



(22) 



If * = 0, then A(^) = Z and Equation (22) is true. We verify 
Equation (22) by induction. Assume that Equation (22) is true for 
ty. Then by adding a pair (Aj,/ij) in & z but not in we obtain a 
subset = ^ U {(Aj,/%)}. 

By (Aj,yUj) ^ we may assume 

r ! (A(f)) = 

and then 

^(AC*!)) 



P'vP'z 



a m i a m +l 



2- 



A;— l' A' 



0' 



2: 



J Pi— It a i' 



■ & 



Now let us consider the skew diagrams obtained by the differences of 
partitions in £ _1 (A(\I/)) and £ _1 (A(\l/i)) respectively, which are the 
same expect the following two rows 



PU 



ct; 



PLi Pi - ai 



a 



ai - P[_ 



Pi 



and 



l (A(^)): fr_l 



ai - P'i_ x 


Pi - ai 


a'i+i - Pi 


ai - P[_i 


Pi - ai 





Here the boxes are the two rows of the skew diagrams associated with 
£ _1 (A(\I/)) and £ -1 (A(\l/i)), and the integers in the boxes are the 
parts corresponding to the differences of partitions in £ -1 (A(\l/)) and 
C~ l {A{^i)). The integers on the left of the boxes are the parts of the 
smaller partitions in £ _1 (A(\I/)) and £ -1 (A(\l/i)). By the assumption 
on £ _1 (A(\I/)) and ai < Pi, the skew diagram of £ _1 (A(\l/i)) is also in 



and 



K/r 1 ^)))! = 

Then by Lemma 4.1 



\v(£- 1 (A(y)))\+2(p i -a l ). 



W 2 n 



'_-j\|¥n$z|+/9 i -a i 



2(» 



In addition, by the definition of $ z , (-1)1*^1 = (_i)l*n*z[+ft-oi_ 
Therefore 

<S BI £- 1 (A(* 1 ))> WaB = (-l)l^ n *'l. 

On the other hand, if there is a partition (^,) in T^, one may reverse 
the previous operation (i.e. removing pairs) and obtain a partition in 

c-\s n ). 

In sum, Ylz&s n c (^' z ) ^ s a summan d of S n , and every irre- 
ducible V^n-module in E n is in ^2 z&Sn c(Z, $, <E> Z ) with the same sig- 
nature. Then Equation (21) follows. 

Let d be the number of singles in Z. By (H n , c(Z, $ z , &z))w 2n = 2 d , 

(R{c{Z,$ z ,$ z )),lnd%U) G F = 2 d . 

In addition, (R(c(Z, $ z , $ z )), R{c{Z, $ z , $ z ))) g f = 2 d . By Theo- 
rem 1.1, every unipotent representation in R(c(Z, <&z, $z)) is distin- 
guished. This completes the theorem. □ 

Example 4.3. Let G F = Sp 4 (Fq). We continue Example 2.2. In this 
case, S x = {(!), (°{ 2 )}- Let Z = (°; 2 ) and <5> z = {(0,1)}. By (2) and 
Theorem 4-2, 

«(i,< : ^ ) i) = i + p( 2 1 ) + ^ 

Example 4.4. Let G F = Sp 12 and 

Lei Z = (°' 5 ) and $ z = {(0, 2)} and <f> z = 0. We /*awe 

^^e))-(° 2 5 ) + K 2 o 5 )-( 2 5 ) + <° 5 2 )- 

Hence these two unipotent representations are distinguished. 
IfZ= then § z = {(0, 1), (2,3)} and § z = $z- We have 

The decomposition of R(c(Z,$ z ,$ z )) is given in (3) and each con- 
stituent is distinguished. 
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